Multisensory Experiences: Where the Senses Meet Technology

Multisensory experiences, that is, experiences that involve more than one of our senses, are part of our everyday life. We often tend to take them for granted, at least when our different senses function normally (normal sight functioning) or are corrected-to-normal (using glasses). However, closer inspection to any, even the most mundane experiences, reveals the remarkable sensory world in which we live in. While we have built tools, experiences and computing systems that have played to the human advantages of hearing and sight (e.g., signage, modes of communication, visual and musical arts, theatre, cinema and media), we have long neglected the opportunities around touch, taste, or smell as interface/interaction modalities. Within this keynote I will share my vision for the future of computing/HCI and what role touch, taste, and smell can play in it

Rapid biodiversity assessment of arthropods for monitoring average local species richness and related ecosystem services

Rapid biodiversity assessment (RBA) is proposed as an affordable indicator for monitoring local species richness of arthropods and sustainability of related ecosystem services. The indicator is based on strictly standardised sampling procedures and the identification of parataxonomic units (morphospecies) instead of species identification. The collection of arthropods was optimized with regard to trap types, time and length of collecting period, selection of four out of seven weekly samples, and choice of counted taxa and trophic guilds. By measuring arthropod activity, RBA is an indicator for functional diversity. Over a period of 8years, average yearly numbers of morphospecies were assessed in Switzerland in 15 agricultural habitats, 15 managed forests, and in 12 unmanaged habitats ranging from protected lowland wetlands to Alpine meadows. The yearly RBA-trend in unmanaged habitats is used for assessing the influence of climate and weather on biodiversity, and as a reference for measuring the relative influences of recent management changes in agriculture and forestry. The average number of morphospecies per sampling station per year depends on temperature, and was only marginally significantly increasing over time in agriculture, but not in forestry or unmanaged areas. Three RBA indices considered to be relevant for maintaining ecosystem services were calculated from the average number of morphospecies per location per year: (1) indicator for ecological resilience and sustainability (all morphospecies); (2) indicator for pollinator diversity (taxa with a majority of pollinators) and (3) indicator for biocontrol diversity (ratio between carnivore and herbivore guilds

Multisensory Experiences: A Primer

We present a primer on multisensory experiences, the different components of this concept, as well as a reflection of its implications for individuals and society. We define multisensory experiences, illustrate how to understand them, elaborate on the role of technology in such experiences, and present the three laws of multisensory experiences, which can guide discussion on their implications. Further, we introduce the case of multisensory experiences in the context of eating and human-food interaction to illustrate how its components operationalize. We expect that this article provides a first point of contact for those interested in multisensory experiences, as well as multisensory experiences in the context of human-food interaction.</jats:p

The how and why behind a multisensory art display

Designing multisensory experiences has always fascinated artists and scientists alike. In recent years, there has been a growing interest in multisensory experience design within the HCI community [1]. Next to advances in haptic technologies, we see novel work on olfactory and gustatory systems [2,3] and efforts in determining multisensory design spaces [4]. Moreover, artists, museum curators, and creative industries are interested in those emerging technologies for their own work. Here we present Tate Sensorium, a multisensory art display, as an example case for multisensory design

Realizing live sequence charts in SystemVerilog.

The design of an embedded control system starts with an investigation of properties and behaviors of the process evolving within its environment, and an analysis of the requirement for its safety performance. In early stages, system requirements are often specified as scenarios of behavior using sequence charts for different use cases. This specification must be precise, intuitive and expressive enough to capture different aspects of embedded control systems. As a rather rich and useful extension to the classical message sequence charts, live sequence charts (LSC), which provide a rich collection of constructs for specifying both possible and mandatory behaviors, are very suitable for designing an embedded control system. However, it is not a trivial task to realize a high-level design model in executable program codes effectively and correctly. This paper tackles the challenging task by providing a mapping algorithm to automatically synthesize SystemVerilog programs from given LSC specifications

I'm sensing in the rain: Spatial incongruity in visual-tactile mid-air stimulation can elicit ownership in VR users

Major virtual reality (VR) companies are trying to enhance the sense of immersion in virtual environments by implementing haptic feedback in their systems (e.g., Oculus Touch). It is known that tactile stimulation adds realism to a virtual environment. In addition, when users are not limited by wearing any attachments (e.g., gloves), it is even possible to create more immersive experiences. Mid-air haptic technology provides contactless haptic feedback and offers the potential for creating such immersive VR experiences. However, one of the limitations of mid-air haptics resides in the need for freehand tracking systems (e.g., Leap Motion) to deliver tactile feedback to the user's hand. These tracking systems are not accurate, limiting designers capability of delivering spatially precise tactile stimulation. Here, we investigated an alternative way to convey incongruent visual-tactile stimulation that can be used to create the illusion of a congruent visual-tactile experience, while participants experience the phenomenon of the rubber hand illusion in VR

OWidgets: A toolkit to enable smell-based experience design

Interactive technologies are transforming the ways in which people experience, interact and share information. Advances in technology have made it possible to generate real and virtual environments with breath-taking graphics and high-fidelity audio. However, without stimulating the other senses such as touch and smell, and even taste in some cases, such experiences feel hollow and fictitious; they lack realism. One of the main stumbling blocks for progress towards creating truly compelling multisensory experiences is the lack of appropriate tools and guidance for designing beyond audio-visual applications. Here we focus particularly on the sense of smell and how smell-based design can be enabled to create novel user experiences. We present a design toolkit for smell (i.e., OWidgets). The toolkit consists of a graphical user interface and the underlying software framework. The framework uses two main components: a Mapper and Scheduler facilitating the device-independent replication of olfactory experiences. We discuss how our toolkit reduces the complexity of designing with smell and enables a creative exploration based on specific design features. We conclude by reflecting on future directions to extend the toolkit and integrate it into the wider audio-visual ecosystem

I Smell Trouble: Using Multiple Scents To Convey Driving-Relevant Information

Cars provide drivers with task-related information (e.g. "Fill gas") mainly using visual and auditory stimuli. However, those stimuli may distract or overwhelm the driver, causing unnecessary stress. Here, we propose olfactory stimulation as a novel feedback modality to support the perception of visual notifications, reducing the visual demand of the driver. Based on previous research, we explore the application of the scents of lavender, peppermint, and lemon to convey three driving-relevant messages (i.e. "Slow down", "Short inter-vehicle distance", "Lane departure"). Our paper is the first to demonstrate the application of olfactory conditioning in the context of driving and to explore how multiple olfactory notifications change the driving behaviour. Our findings demonstrate that olfactory notifications are perceived as less distracting, more comfortable, and more helpful than visual notifications. Drivers also make less driving mistakes when exposed to olfactory notifications. We discuss how these findings inform the design of future in-car user interfaces

“Sweet: I did it”! Measuring the sense of agency in gustatory interfaces

Novel gustatory interfaces offer the potential to use the sense of taste as a feedback modality during the interaction. They are being explored in a wide range of implementations, from chemical to electrical and thermal stimulation of taste. However, the fundamental aspect of gustatory interaction that has yet to be explored is the Sense of Agency (SoA). It is the subjective experience of voluntary control over actions in the external world. This work investigates the SoA in gustatory systems using the intentional binding paradigm to quantify how different taste outcome modalities influence users' SoA. We first investigate such gustatory systems using the intentional binding paradigm to quantify how different tastes influence users' SoA (Experiment 1). The gustatory stimuli were sweet (sucrose 75.31 mg/ml), bitter (caffeine powder 0.97 mg/ml), and neutral (mineral water) as the outcomes of specific keyboard presses. We then investigated how SoA was altered depending on users' sweet liking phenotype, given that sweet is one of the taste outcomes (Experiment 2), and in contrast with audio as a traditional outcome. In Experiment 2, stronger taste concentrations (sweet-sucrose 342.30 g/L, bitter-quinine 0.1 g/L, and neutral) were used, with only participants being moderate sweet likers. We further contrasted tastes with audio as the traditional outcome. Our findings show that all three taste outcomes exhibit similar intentional binding compared to auditory in medium sweet likers. We also show that longer action-outcome duration improved the SoA. We finally discuss our findings and identify design opportunities considering SoA for gustatory interfaces and multisensory interaction

The Scent of Collaboration: Exploring the Efect of Smell on Social Interactions

Social interactions are multisensory experiences. However, it is not well understood how technology-mediated smell can support social interactions, especially in collaborative tasks. To explore its effect on collaboration, we asked eleven pairs of users to work together on a writing task while wearing an interactive jewellery designed to emit scent in a controlled fashion. In a within-subjects experiment, participants were asked to collaboratively write a story about a standardized visual stimulus while exposed to with scent and without scent conditions. We analyzed video recordings and written stories using a combination of methods from HCI, psychology, sociology, and human communication research. We observed differences in both participants' communication and creation of insightful stories in the with scent condition. Furthermore, scent helped participants recover from communication breakdown even though they were unaware of it. We discuss the possible implications of our findings and the potential of technology-mediated scent for collaborative activities